1. Technical Field
Example embodiments of the present invention relate in general to a method and apparatus for evaluating the quality of software, and more specifically, to fault-based software testing technology for evaluating the quality of software through fault injection. In particular, examples of the present invention relate in general to a fault-based software testing method and system capable of reducing time and cost for software testing.
2. Related Art
Fault-based testing refers to a method that selects test data, focusing on finding types of errors, based on the predictive knowledge that the types of errors having occurred frequently in the past are likely for the system, system components or programs under testing.
Such fault-based testing may be applied to software engineering. Fault-based testing for a developed program (testing target program) is performed by generating a plurality of error programs through artificial fault injection with respect to the testing target program, and executing the generated error programs by using test data.
FIG. 1 is a block diagram illustrating a general configuration of a conventional fault-based software testing system.
Referring to FIG. 1, the conventional fault-based software testing system 100 may include an error program generating unit 110 and a testing unit 120.
The error program generating unit 110 receives a testing target program and an error type list, analyzes the testing target program, and generates errors corresponding to the received error type list.
In this case, types of errors mean the types of errors predicted to occur frequently in a system. In the case of software, examples of the types of frequent errors include wrong use of an arithmetic operator (for example, wrong use of an arithmetic operator ‘−’ instead of an arithmetic operator ‘+’), wrong use of a logical operator (for example, wrong use of a logical operator ‘≧’ instead of a logical operator ‘>’), and the like.
The results of the error program generating unit 110 are mutants resulting from the modification of the testing target program according to the given types of errors. FIG. 1 illustrates a case where M error programs, including an error program #1, an error program #2, . . . , an error program #M, are generated.
The generated error programs are executed through the testing unit 120. The error programs are executed using a testing set, which is a set of M test data, until errors caused by codes modified by the error program generating unit 110 are found.
FIG. 2 is a flowchart illustrating an example of the operation of the testing unit in the fault-based software testing system.
Referring to FIG. 2, the testing unit 120 executes the M error programs by using the testing set including M test data.
First, in step S210, all elements arranged in an error detection flag Error[ ], which represents whether an error is detected in a relevant error program, are set to ‘UNDETECTED’ indicating a state that no error is detected.
Then, using a variable i (i=1, . . . , N) indicating the test data and a variable j (j=1, . . . , M) indicating the error programs, the error programs are executed with respect to the respective test data, and it is determined whether errors by the error programs are detected (S220 and S221).
It is determined whether an error has already been detected in a relevant error program j by other test data (S222). When the error has already been detected in the error program, the execution of the corresponding test data is omitted. The error program j is executed using test data i (S223), and it is determined whether an error is detected (S224). When no error is detected, a next error program ((j+1)th error program) is executed and the process repeats the above steps from step S222. When an error is detected, an error detection flag Error[j] for the error program j is set to ‘DETECTED’ (S225).
The conventional fault-based testing system uses too much time and money on the generation and execution of error programs. For reference, according to the paper, “An experimental determination of sufficient mutation operators,” (A. J. Offutt, Ammei Lee, G. Rothermel, R. Untch and C. Zapf. “A experimental determination of sufficient mutation operators.”, ACM Transaction of Software engineering methodology, 5(2):99-118, April 1996), when the testing method (mutation testing) described in detail with reference FIG. 2 was used for simple program codes of 28 lines, 951 error programs were generated, and the number of error programs generated was rapidly increased as the size of the program became larger.
That is, as in the procedures of the fault-based testing exemplified in FIG. 2, the error programs are repetitively executed until the corresponding errors are detected. In the case of the mutation testing, one of the fault-based testing methods, it takes a long time to repetitively execute the error programs because a large number of error programs are generated.